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Abstract 

We present a technique which complements 
■Hidden Markov Models by incorporating some 



appropriate smoothing techniques show better 
performance than the first order models and is 



loxicalizod states representing syntactically un 



common words. Our approach examines the 
distribution of transitions, selects the uncom- 
mon words, and makes lexicalized states for the 

wnrHs Wp pprfnrmprl a. part-nf-sppprh tagging 



experiment on the Brown corpus to evaluate the 



resultant language model and discovered that 
this technique improved the tagging accuracy 
by 0.21% at the 95% level of confidence. 

1 Introduction 

Hidden Markov Models are widely used for 
statistical language modelling in various fields, 
e.g., part-of-speech tagging or speech recogni- 
tion ( [Rabiner and Juang, 1986 ). The models 
are based on Markov assumptions, which make 
it possible to view the language prediction as a 
Markov process. In general, we make the first- 
order Markov assumptions that the current tag 
is only dependant on the previous tag and that 
the current word is only dependant on the cur- 
rent tag. These are very 'strong' assumptions, 
so that the first-order Hidden Markov Models 
have the advantage of drastically reducing the 
number of its parameters. On the other hand, 
the assumptions restrict the model from utiliz- 
ing enough constraints provided by the local 
context and the resultant model consults only 
a single category as the contex. 

A lot of effort has been devoted in the past 
to make up for the insufficient contextual in- 
formation of the first-order probabilistic model. 
The second order Hidden Markov Models with 
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considered a state-of-the-art technique (Mcri 
aldo, 1994; Brants, 1996| ). The complexity of 



the model is however relatively very high con- 
sidering the small improvement of the perfor- 
mance. 

Garside describes IDIOMTAG QGarside et 



al., 1987) which is a component of a part-of- 
speech tagging system named CLAWS. ID- 
IOMTAG serves as a front-end to the tagger 
and modifies some initially assigned tags in or- 
der to reduce the amount of ambiguity to be 
dealt with by the tagger. IDIOMTAG can 
look at any combination of words and tags, 
with or without intervening words. By using 
the IDIOMTAG, CLAWS system improved tag- 
ging accuracy from 94% to 96-97%. However, 
the manual-intensive process of producing id- 
iom tags is very expensive although IDIOMTAG 
proved fruitful. 



Kupiec ( Kupiec, 1992 ) describes a technique 
of augmenting the Hidden Markov Models for 
part-of-speech tagging by the use of networks. 
Besides the original states representing each 
part-of-speech, the network contains additional 
states to reduce the noun/adjective confusion, 
and to extend the context for predicting past 
participles from preceding auxiliary verbs when 
they are separated by adverbs. By using these 
additional states, the tagging system improved 
the accuracy from 95.7% to 96.0%. However, 
the additional context is chosen by analyzing 
the tagging errors manually. 

An automatic refining technique for Hidden 
Markov Models has been proposed by Brants 
( Brants, 1996| ). It starts with some initial first 
order Markov Model. Some states of the model 
are selected to be split or merged to take into 
account their predecessors. As a result, each of 



new states represents a extended context. With 
this technique, Brants reported a performance 
equivalent to the second order Hidden Markov 
Models. 

In this paper, we present an automatic re- 
fining technique for statistical language models. 
First, we examine the distribution of transitions 
of lexicalized categories. Next, we break out the 
uncommon ones from their categories and make 
new states for them. All processes are auto- 
mated and the user has only to determine the 
extent of the breaking-out. 

2 "Standard" Part-of-Speech 

Tagging Model based on HMM 

From the statistical point of view, the tagging 
problem can be defined as the problem of find- 
ing the proper sequence of categories ci jn = 
ex, C2, c n (n > 1) given the sequence of words 
wi,n = ^1,^2, • w n (We denote the i'th word 
by Wi, and the category assigned to the Wi by 
Cj), which is formally defined by the following 
equation: 

T{wi,n) = arg max P(ci tn \wi in ) (1) 



Charniak flCharniak et al., 1993 ) describes 
the "standard" HMM-based tagging model as 
Equation J2L which is the simplified version of 
Equation |l[ 

n 

T{w\,n) = arg max Y[P(ci\ci-i)P(wi\ci) (2) 



' i=l 

With this model, we select the proper category 
for each word by making use of the contextual 
probabilities, P(cj|cj_i), and the lexical prob- 
abilities, P{wi\Ci). This model has the advan- 
tages of a provided theoretical framework, auto- 
matic learning facility and relatively high per- 
formance. It is thereby at the basis of most tag- 
ging programs created over the last few years. 

For this model, the first-order Markov assum- 
tions are made as follows: 



P(Ci\ci ti -l,Wl^l) ps P(ci\c 



i-lj 



(3) 



P(wi\c lti ,wi^i) ps P{wi\ci) (4) 

With Equation [| we assume that the current 
category is independent of the previous words 
and only dependent on the previous category. 



With Equation f|, we also assume that the cor- 
rect word is independent of everything except 
the knowledge of its category. Through these 
assumptions, the Hidden Markov Models have 
the advantage of drastically reducing the num- 
ber of parameters, thereby alleviating the sparse 
data problem. However, as mentioned above, 
this model consults only a single category as 
context and does not utilize enough constraints 
provided by the local context. 

3 Some Refining Techniques for 
HMM 

The first-order Hidden Markov Models de- 
scribed in the previous section provides only 
a single category as context. Sometimes, this 
first-order context is sufficient to predict the 
following parts-of-speech, but at other times 
(probably much more often) it is insufficient. 
The goal of the work reported here is to develop 
a method that can automatically refine the Hid- 
den Markov Models to produce a more accu- 
rate language model. We start with the care- 
ful observation on the assumptions which are 
made for the "standard" Hidden Markov Mod- 
els. With the Equation ||, we assume that the 
current category is only dependent on the pre- 
ceding category. As we know, it is not always 
true and this first-order Markov assumption re- 
stricts the disambiguation information within 
the first-order context. 

The immediate ways of enriching the context 
are as follows: 

• to lexicalize the context. 

• to extend the context to higher-order. 

To lexicalize the context, we include the pre- 
ceding word into the context. Contextual prob- 
abilities are then defined by P(cj|cj_i, ^i-i). 
Figure |l] illustrates the change of dependency 
when each method is applied respectively. Fig- 
ure [j](a) represents that each first-order contex- 
tual probability and lexical probability are in- 
dependent of each other in the "standard" Hid- 
den Markov Models, where Figure |](b) repre- 
sents that the lexical probability of the preced- 
ing word and the contextual probability of the 
current category are tied into a lexicalized con- 
textual probability. 

To extend the context to higher-order, we ex- 
tend the contextual probability to the second- 



(a) 




(b) 




Figure 1: Two Types of Weakening the Markov 
Assumption 



order. Contextual probabilities are then defined 
by P(ci\ci-i, Ci-2). Figure |l](c) represents that 
the two adjacent contextual probabilities are 
tied into the second-order contextual probabil- 
ity. 

The simple way of enriching the context is to 
extend or lexicalize it uniformly. The uniform 
extension of context to the second order is fea- 
sible with an appropriate smoothing technique 
and is considered a state-of-the-art technique, 
though its complexity is very high: In the case 
of the Brown corpus, we need trigrams up to 
the number of 0.6 million. An alternative to the 
uniform extension of context is the selective ex- 
tension of context. Brants ( Brants, 1996] ) takes 
this approach and reports a performance equiv- 
alent to the uniform extension with relatively 
much low complexity of the model. 

The uniform lexicalization of context is com- 
putationally prohibitively expensive: In the 
case of the Brown corpus, we need lexicalized 
bigrams up to the number of almost 3 billion. 
Moreover, many of these bigrams neither con- 
tribute to the performance of the model, nor oc- 
cur frequently enough to be estimated properly. 
An alternative to the uniform lexicalization is 
the selective lexicalization of context, which is 
the main topic of this paper. 



4 Selective Lexicalization of HMM 

This section describes a new technique for re- 
fining the Hidden Markov Model, which we call 
selective lexicalization. Our approach automat- 
ically finds out syntactically uncommon words 



and makes a new state (we call it a lexicalized 
state)for each of the words. 

Given a fixed set of categories, {c l ,c 2 , ...,c c '}, 
e.g., {adjective, verb}, we assume the dis- 
crete random variable X cJ with domain the set 
of categories and range a set of conditional prob- 
abilities. The random variable X c j then repre- 
sents a process of assigning a conditional prob- 
ability -P(c l |e') to every category c l (c l ranges 
over c l ...c c ) 



X 
X 



C J ' 



C 3\ 



P(cV') 
P(c 2 \ci) 



X cJ (c c ) =P{c c \ci) 

We convert the process of X cJ into the state 
transition vector, V cJ , which consists of the 
corresponding conditional probabilities, e.g., 

V prep = (P(adjective\prep), P(verb\prep)) T . 

The (squared) distance between two arbitrary 
vectors is then computed as follows: 

«(Vi,V 2 ) = (V 1 -V 2 ) T (V 1 -V 2 ) (5) 

Similarly, we define the lexicalized state 
transition vector^, Y c j w k, e.g., 

(P(adjective\prep, in), P(verb\prep, in)) T . 

In this situation, it is possible to regard each 
lexicalized state transition vector, V cJ w k, of the 
same category c J as members of a cluster whose 
centroid is the state transition vector, V cJ . We 
can then compute the deviation of each lexi- 

from its 



calized state transition vector 
corresponding centroid. 



5 "^C^ ) 



D(V 



C J ,W k J 



(V 



C-? ,w k 



-v c ,f(v 



(6) 



Figure [2] represents the distribution of lexical- 
ized state transition vectors according to their 
deviations. As you can see in the figure, the 
majority of the vectors are near their centroids 
and only a small number of vectors are very far 
from their centroids. In the first-order context 



1 To alleviate the sparse data problem, we smoothed 
the lexical ized state transition proba bilities by MacKay 
and Peto( viacKay and Peto, lOQ^ 's smoothing tech- 
nique. 
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Figure 2: Distribution of Lexicalized Vectors according to Deviation 



model (without considering lexicalized context), 
the centroids represent all the members belong- 
ing to it. In fact, the deviation of a vector is a 
kind of (squared) error for the vector. The error 
for a cluster is 

e(V c3 )=]>XV cW ) (7) 

w k 

and the error for the overall model is simply the 
sum of the individual cluster errors: 

£ = ]Te(V cJ ) (8) 

cJ 

Now, we could break out a few lexicalized 
state vectors which have large deviation {D > 9) 
and make them individual clusters to reduce the 
error of the given model. 

As an example, let's consider the preposi- 
tion cluster. The value of each component 
of the centroid, V prep , is illustrated in Figure 
||(a) and that of the lexicalized vectors, Vp rep i n , 
■ep,with and Vprep, ou t are in Figure |3](b), (c) 
and (d) respectively. As you can see in these fig- 
ures, most of the prepositions including in and 
with are immediately followed by article(AT), 
noun(NN) or pronoun(NP), but the word out as 
preposition shows a completely different distri- 
bution. Therefore, it would be a good choice to 
break out the lexicalized vector, V pre p,out, from 
its centroid, V p . rep . 



out 




(a) (b) 



Figure 4: Splitting the preposition State 



From the viewpoint of a network, the state 
representing preposition is split into two 
states; the one is the state representing ordi- 
nary prepositions except out, and the other is 
the state representing the special preposition 
out, which we call a lexicalized state. This 
process of splitting is illustrated in Figure ||. 

Splitting a state results in some changes of 
the parameters. The changes of the parame- 
ters resulting from lexicalizing a word, w k , in a 
category, c* , are indicated in Table [j] (c* ranges 
over c^...c ). This full splitting will increase 
the complexity of the model rapidly, so that 
estimating the parameters may suffer from the 
sparseness of the data. 

To alleviate it, we use the pseudo splitting 



1 



Ttti i 1 1T1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1T1 1 1 1 1 1 1 1 itti 1 1 1 1 iti 1 1 1 1T1 1 1 1 1 1 1 1 1 1T1 1 1 1 1 1 1 1 1 1 1 1T1 1 1 1 1 1 1 

NP AT NN PREP 

(a) preposition 



Ttti i i iTi i i i i i i i i i i i iti iTi i i i i i i i iIti i i i i i i i i i i iti 1 1 1 1 1 1 1 1 1T1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 

NP AT NN PREP 

(b) in as preposition 
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(c) with as preposition 




ITU 1 1 ITI 1 1 1 1 III I I 1 1 ITI III I I ITI 1 1 rrn MM III I 1 1 III I 1 1 I 1 1 ititi I ITI 1 1 1 1 1 1 ITI 1 1 1 1 1 1 

NP AT NN PREP 

(d) out as preposition 



Figure 3: Transition Vectors in preposition Cluster 



Table 1: Changes of Parameters in Full Split- 
ting 



before splitting 


after splitting 


P(w % \o>) 


P(w l \c j ,w k ) 
P{w l \d> ,^w k ) 


P(cV) 


P{c l \c>,w k ) 
P(c i \c^,^w k ) 


P(c>\c l ) 


P(c 1 ,w k \c l ) 

P(c?,^W k \<*) 



which leads to relatively small increment of the 
parameters. The changes of the parameters in 
pseudo splitting are indicated in Table 

5 Experimental Result 

We have tested our technique through part-of- 
speech tagging experiments with the Hidden 
Markov Models which are variously lexicalized. 
In order to conduct the tagging experiments, we 
divided the whole Brown (tagged) corpus con- 
taining 53,887 sentences (1,113,191 words) into 



Table 2: Changes of Parameters in Pseudo 
Splitting 



before splitting 


after splitting 


P(u-V) 


P(u-V) 


P(cV) 


P(c l \c>,w k ) 
P(cV',-^ fe ) 


P{c>\c l ) 


P{c>\c l ) 



two parts. For the training set, 90% of the 
sentences were chosen at random, from which 
we collected all of the statistical data. We re- 
served the other 10% for testing. Table [|| lists 
the basic statistics of our corpus. 



Table 3: Overview of Our Corpora 





# of sentences 


$ of words 


training set 


48,499 


1,001,712 


test set 


5,388 


111,479 



We used a tag set containing 85 categories. 
The amount of ambiguity of the test set is sum- 
marized in Table |||. The second column shows 
that words to the ratio of 52% (the number 
of 57,808) are not ambiguous. The tagger at- 
tempts to resolve the ambiguity of the remain- 
ing words. 

Table 4: Amount of Ambiguity of Test Set 



ambiguity(#) 


1 


2 


3 


4 


5 


6 


total 


ratio(%) 


52 


30 


8 


7 


2 


1 


100 



Figure [5] and Figure |6| show the results of 
our part-of-speech tagging experiments with the 
"standard" Hidden Markov Model and vari- 
ously lexicalized Hidden Markov Models us- 
ing full splitting method and pseudo splitting 
method respectively. 

We got 95.7858% of the tags correct when 
we applied the standard Hidden Markov Model 
without any lexicalized states. As the num- 
ber of lexicalized states increases, the tagging 
accuracy increases until the number of lexical- 
ized states becomes 160 (using full splitting) 
and 210 (using pseudo splitting). As you can 
see in these figures, the full splitting improves 
the performance of the model more rapidly 
but suffer more sevelery from the sparseness 
of the training data. In this experiment, we 
employed Mackay and Peto's smoothing tech- 
niques for estimating the parameters required 
for the models. The best precision has been 
found to be 95.9966% through the model with 
the 210 lexcalized states using the pseudo split- 
ting method. 

6 Conclusion 

In this paper, we present a method for comple- 
menting the Hidden Markov Models. With this 
method, we lexicalize the Hidden Markov Model 
seletively and automatically by examining the 
transition distribution of each state relating to 
certain words. 

Experimental results showed that the selec- 
tive lexicalization improved the tagging accu- 
rary from about 95.79% to about 96.00%. Using 
normal tests for statistical significance we found 
that the improvement is significant at the 95% 
level of confidence. 



The cost for this improvement is minimal. 
The resulting network contains 210 additional 
lexicalized states which are found automati- 
cally. Moreover, the lexicalization will not de- 
crease the tagging speedQ, because the lexi- 
calized states and their corresponding original 
states are exclusive in our lexicalized network, 
and thus the rate of ambiguity is not increased 
even if the lexicalized states are included. 

Our approach leaves much room for improve- 
ment. We have so far considered only the outgo- 
ing transitions from the target states. As a re- 
sult, we have discriminated only the words with 
right-associativity. We could also discriminate 
the words with left-associativity by examining 
the incoming transitions to the state. Further- 
more, we could extend the context by using the 
second-order context as represented in Figure 
H(c). We believe that the same technique pre- 
sented in this paper could be applied to the pro- 
posed extensions. 
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Figure 5: POS tagging results with lexicalized HMM using full splitting method 



0.96 



0.9595 



0.959 



0.9585 



0.958 



0.9575 




100 150 
# of lexicalized states 



250 



Figure 6: POS tagging results with lexicalized HMM using pseudo splitting method 
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Appendix : 

Top 100 words with high deviation 

according(IN) rather(IN) out(IN) able(JJ) 
Aj(NP) no(QL) because(RB) however(RB) 
as(IN) per(IN) trying(VBG) however(WRB) 
likely(JJ) United(NP) Mrs.(NP) New(NP) 
Rhode(NP) National(NP) Miss(NP) 
tried(VBD) Dr.(NP) lack(NN) much(QL) 
Mr.(NP) North(NP) June(NP) A.(NP) 
J.(NP) right(QL) May(NP) ready(JJ) 



St.(NP) even(QL) various(JJ) don't(DO) 
instead(RB) far(QL) B(NP) didn't(DOD) 
try(VB) available(JJ) William(NP) !(.) ?(.) 
;(.) number(NN) so(CS) due(JJ) World(NP) 
Christian(NP) difficult(JJ) tell(VB) go- 
ing(VBG) kind(NN) let(VB) continue(VB) 
series(NN) part(NN) radio(NN) sure(JJ) 
want(VB) front(NN) seem(VB) total(NN) 
decided(VBD) expected(VBN) right(NN) 
based(VBN) White(NP) except(IN) told(VBD) 
James(NP) fact(NN) March(NP) sort(NN) 
example(NN) designed(VBN) respect(NN) 
talk(VB) Department(NP) single(AP) Nc- 
gro(NP) wanted(VBD) Western(NP) yes(RB) 
become(VBN) necessary(JJ) speak(VB) 
about(RB) amount(NN) down(IN) like(VB) 
S.(NP) same(AP) too(RB) General(NP) 
began(VBD) use(NN) tax(NN) got(VBN) 



